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EXECUTIVE SUMMARY 


INTRODUCTION 

Minnesotans value education. Education is the largest financial investment in the state budget, and those investments 
place Minnesota students among the top performers in the nation. Still, substantial gaps in opportunity and 
performance persist. 

The three purposes of the needs assessment were to: 

1. Describe infrastructure, capacity, and costs of program evaluation and educational assessment; 

2. Identify unmet needs; and 

3. Evaluate how CAREI might contribute a solution. 

This report summarized the findings of our study. It also provides a brief summary of how the Center for Applied 
Research and Educational Improvement (CAREI) might be used to network and leverage personnel, expertise, and 
resources to solve pressing problems. CAREI can contribute to expand infrastructure and capacities that meet the needs 
of education in the state with a high return on investment. 

KEY FINDINGS AND RESULTS 

The conclusions from our 2015 statewide needs assessment are very clear. There are substantial unmet needs at all 
levels of the Minnesota educational systems with respect to research, evaluation, assessment, and data use. CAREI 
initiated this needs assessment in February of 2015. An initial draft of the report was made available for public 
comment in November, and the report was finalized and distributed in December of 2015. 

Researchers and evaluators from CAREI interviewed and surveyed educational leaders and other stakeholders. For the 
purpose of the needs assessment, CAREI gathered input from approximately 800 individuals and 13 professional 
organizations. The following are some key findings. 

• A large majority of respondents believe good quality data can improve educational decisions. Nearly everyone 
with whom we spoke articulated that they would benefit from additional assistance in the use of data. 

• A large percentage of survey respondents indicated their school's or district's capacity to effectively use data to 
guide educational decisions was fair or poor. Despite substantial motivations and efforts to use data, most 
educational systems in Minnesota lack the capacity to meet their own needs for data-based decision making. 

• Educational professionals consistently described the need for additional resources to help them use data. There 
are very limited resources at all levels of the educational systems in Minnesota to access, analyze, present, and 
interpret data in a timely and effective manner. 

• Respondents consistently described the need for professional development and expert consultation to assist, 
automate, and deliver timely reports and provide guidance for the use of data for educational decisions. 
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• Educational leaders indicated that they do not have the time or personnel to design, develop, or implement 
assessment and evaluation programs that would effectively inform leadership and practice. The results of 
surveys indicate that those professionals are too few in number in Minnesota and, perhaps, too costly for many 
educational systems to afford. 

• Relatively few school districts in Minnesota employ professionals with advanced training in research, program 
evaluation, or assessment/measurement. Those are generally the largest districts concentrated in the metro 
region such that most rural and out-state systems with limited capacity and expertise. 

CONCLUSIONS 


1] BUILD THE INFRASTRUCTURE AND CAPACITY AT REASONABLE COSTS 

Like many other states, Minnesota has established rigorous systems for accountability, which include state standards. It 
has also established useful programs for data collection. Those include the Minnesota Comprehensive Assessments 
(MCAs), the Minnesota Student Survey, and the Minnesota Statewide Longitudinal Education Data System (SLEDS). The 
returns on those and other investments have yet to be realized, in part, because there are substantial unmet needs for 
educational systems and professionals to use them. Additional supports may substantial improve their uses by policy 
makers, educational leaders, and educators. 

Those who responded to the interviews and surveys consistently indicated a lack of resources and expertise to support 
their efforts. To truly leverage state and local investments, professional educators require infrastructure to build 
capacity and efficiencies to use data that improve educational outcomes. 

2) MEET THE UNMET NEEDS 

The results indicate the following are some of the unmet needs throughout the state. The needs were most substantially 
unmet in rural and other high need communities. 

• Infrastructure: Educational networks for data use and school improvement are an unmet need in many areas of 
the state. Public and centralized services are needed to identify common challenges among districts and 
coordinate efforts to implement solutions. 

• Capacity: Districts have limited capacity to provide uniform and high quality research, assessment, and 
evaluation services. These services improve outcomes only when they are done well and too few districts have 
the necessary personnel and resources. 

• Efficiency: It is too costly for most districts to hire the necessary personnel and invest the necessary resources to 
efficiently and effectively support data-based decision making. The lack of centralized services to support those 
practices throughout the state represents an unmet need that limits the potential of Minnesota's educational 
systems. 

• Time: Data are often difficult to access and require time to prepare before the data can guide educational 
decisions. Leaders, educators and systems of education need others to do this time-intensive work and they 
need it done more efficiently. The analysis and reports should be available soon after data collection, not 
months later when interest and opportunities to use the data have passed. 
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• Ease of use: It is hard to use data to inform decisions without sufficient supports. Ease of use is an unmet need. 
Preparation, analysis, and reporting of data are difficult when they are not centralized or automated. Because 
they are not, each educational agency must invest personnel time to prepare data, automate analysis, refine 
methods, and build reports. 

• Training: There is a need for professional development to improve data literacy (i.e., analyze, display, interpret, 
and use data). Too few educators are well-prepared to collect and use data systematically and effectively. 

• Expertise and personnel: The Minnesota Department of Education and local educational agencies struggle to 
recruit, retain, and afford staff with the required training and expertise within the domains of research, 
evaluation, assessment, and data use. There is a substantial need and potential benefit if the state can network 
those experts with policy makers, educational leaders, and educators. 


3) LEVERAGE CAREI AND THE UNIVERSITY: NETWORKS OF IMPROVEMENT 

Although there are already investments and infrastructure that establish rigorous standards and accountability, there 
are not complementary investments and infrastructure for agencies and professionals to learn and improve through the 
use of data. Data and standards for accountability are ineffective in lieu of the capacity to change. As was described to 
us by almost all of those interviewed and surveyed, professional educators believe in data, rigor, and improvement. 
There are simply too many unmet needs that limit their capacity to improve. Progress is unlikely without enhanced 
capacity. 

There is an opportunity to use CAREI and the University as a solution. The most scalable and cost efficient solution is to 
leverage the existing expertise and resources at CAREI to build enhanced capacity throughout the state. CAREI can 
establish cost effective and highly efficient Networked Improvement Communities (NICs) 1,2 that are comprised of policy 
makers, the Minnesota Department of Education, districts, schools, and educational professionals. Efforts will be 
organized through voluntary participation in specific problem-focused collaboratives that arise from needs throughout 
the state. For example, collaborative might emerge for early childhood education, technology initiatives, benchmark 
assessments, standards-based grading, or the achievement gap. CAREI will build and maintain the infrastructure, 
services, and processes for shared and equitable capacity throughout the state. 

In its 25 th year of service, CAREI is well-positioned to address many of the unmet needs with respect to networks and 
services for research, evaluation, assessment, and data use. A brief description of this proposed solution is presented 
at the conclusion of this report (p. 15). 


1 Bryk, A. S., Gomez, L. M., Grunow, A., & Lemahieu, P. G. (2015). Learning to Improve: How America's Schools Can Get Better at Getting Better. 
Harvard Education Press: Massachusetts. 

2 Bryk, A.S., Gomez, L.M., Grunow, A. (2010). Stanford, CA: Getting Ideas into Action: Building Networked Improvement Communities in education. 
Carnegie Foundation for the Advancement of Teaching, http://www.carnegiefoundation.org/resources/publications/getting-ideas-action-building- 
networked-improvement-communities-education/ 
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INTRODUCTION 


ABOUT CAREI 

CAREI was established in 1988 as an independent college-wide center at the University of Minnesota to conduct 
rigorous, unbiased evaluation of innovative practices and policies in education. CAREI has completed more than 250 
evaluation and research studies funded by federal and state agencies, local educational agencies (i.e., school districts), 
and foundations in excess of $30 million. CAREI's staff includes Project Leads with PhDs in relevant and diverse domains. 
Those include educational psychology, statistics and measurement, curriculum and instruction, educational policy and 
administration, educational leadership and evaluation studies. Those Project Leads coordinate research teams and 
collaborate with content experts and community members. 

RESEARCH, DEVELOPMENT AND ENGAGEMENT TO IMPROVE EDUCATION 

The New CAREI is invested in Research, Development and Engagement to Improve Education. In its 25 th year, 

CAREI has an expanded mission (see below) and new leadership: 

• Dr. Theodore J Christ, Director; 

• Dr. Delia Kundin, Associate Director for Evaluation Services; 

• Dr. Kimberly Gibbons, Associate Director for Innovation and Outreach; and 

• Dr. Danielle Dupuis, Assistant Director for Research and Assessment Services. 

In addition, CAREI is comprised of a team of 11 Project Leaders (10 PhD, 1 MA) and a large team of Research 
Assistants. 

There are substantial state and federal resources invested at CAREI and at the University of Minnesota. That provides 
CAREI access to equipment, facilities, technology support, libraries, and an extensive network of University-based 
researchers and experts. There are many faculty at the University who consult with and guide educational agencies 
across the country and around the world. Their expertise and value is widely sought. These experts live right here in 
Minnesota. Their children go to our schools. They value our communities. They are both invested and motivated to 
serve Minnesota policy makers, leaders, educators and agencies. 

There is substantial potential for CAREI contribute innovative solutions to education in Minnesota, but we must first 
identify the needs. To that end, we engaged in this needs assessment to determine what policy makers, state education 
agencies, local educational agencies, schools, and educational professionals would like from CAREI. 


CARETS MISSION 

CAREI's mission is to improve the quality of education for all learners, and thereby society as a whole, through research, 
development, and engagement. As applied researchers and evaluators, we believe that we can have an immediate 
impact on our communities. Accordingly, we listen to and work with our clients and partners to understand their 
experiences. 
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CARETS DIVERSITY STATEMENT 

Individuals experience the world differently because they come from different places and have varied experiences. At 
CAREI, we recognize that all people have strengths, skills, and challenges, so we work to understand these differences. 

Our work requires us to actively reflect on how our methods and interpretations of data may perpetuate privilege and 
prevailing assumptions. We strive to prevent overgeneralization of majority culture and beliefs, especially when these 
beliefs bump up against our diverse communities, cultures, genders, races, ethnicities, (dis)abilities, lifestyles, and 
contexts. 


EDUCATIONAL OUTCOMES FOR STUDENTS IN MINNESOTA 


In June 2015, policy makers in Minnesota approved a two-year, $17 billion 
education budget. That budget invests in early learning initiatives and 
reading tutors along with targeted programs for American Indian and 
second language English learners. Many programs are intended to raise 
the rate of graduation, improve career and college readiness, and close the 
achievement gap. In addition, the per pupil revenue will increase to 
$12,533 by 2017 when general unweighted pupil revenue, food service, 
and debt service aid are combined. 3 

Minnesotans value education. Education is the most substantial investment in 
the state budget, and those investments place Minnesota students among the 
top performers in the nation; however, substantial gaps in opportunity and 
performance persist between White students and students of color. 4 Although less well publicized, there are also 
notable gaps for students in poverty, non-native English language learners, and students with disabilities. 5 These 
performance gaps are observed on the Minnesota Comprehensive Assessments (MCAs), the ACT college admissions test, 
and graduation rates. They are some of the most substantial gaps in the country 6 and many of our children fail to thrive 
or meet the rigorous standards 7 we set. 

On the 2015 MCAs, 59.5% of students were proficient in reading and 60.2% were proficient in math. 8 While many 
Minnesota students met grade level proficiency standards, approximately 40% did not meet standards for proficiency in 



3 Office of the Governor Blog. (2015, July 1). New Investments in E-12 Education Take Effect Today, http://mn.gov/governor/blog/the-office-of-the- 
governor-blog-entrv-detail.isp?id=102-166807. 

4 http://www.education.state.mn.us/MDE/Welcome/OfficeCom/CloseAchievGap/index.html 

5 See http://www.education.state.mn.us/MDE/Welcome/OfficeCom/CloseAchievGap/056710 for more information on achievement gaps between 
students receiving free and reduced price lunch, special education students, and English Language Learners and more advantaged peers. 

6 http://www.startribune.com/minnesota-s-achievement-gap-debated-at-nabi-conference/321150101/; 

http://www.education.state.mn.us/MDE/Welcome/QfficeCom/CloseAchievGap/056710 ; http://minnesota.cbslocal.com/2013/08/21/good- 
question-why-such-a-large-achievement-gap-in-mn-classrooms/; http://www.startribune.com/minnesota-fourth-graders-see-drop-in-math- 

reading-scores-on-national-exam/337784361/ 

7 For information on MN's K-12 academic standards, see http://education.state.mn.us/mde/edexc/stancurri/k-12academicstandards/. 

8 MDE Report Card: http://rc.education.state.mn.us/ 



CENTER FOR APPLIED RESEARCH 
AND EDUCATIONAL IMPROVEMENT 
UNIVERSITY OF MINNESOTA 


Page | 2 



reading or math. Proficiency rates were less among students of color (see Figure 1 for information on Minnesota's 
statewide reading proficiency by race/ethnic group; see Figure 1, p. 3). There are substantial gaps in opportunity and 
proficiency in reading and math across ethic groups. 


MIND THE GAP: 

READING PROFICIENCY IN MINNESOTA 
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Figure 1. Minnesota Statewide Reading, Percent Proficient by Race/Ethnic Group 9 


MIND THE GAP: 

MATH PROFICIENCY IN MINNESOTA 
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Figure 2. Minnesota Statewide Math, Percent Proficient by Race/Ethnic Group 10 


9 http://rc.education.state.mn. us/# 


10 http://rc.education.state.mn.us/# 
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The average ACT composite score in 2015 for White students was 23.7. It was 17.6 for Black/African American students, 
19.3 for American Indian/Alaska Native students, 19.8 for Hispanic/Latino students, 20.7 for Asian students, and 20.1 for 
Native Hawaiian/Other Pacific Islander students. 11 In 2014, 86% of White students graduated as compared to 51% of 
American Indian/Alaskan Native students, 60% of Black students, 63% of Hispanic students, and 87% of Asian students. 12 
If schools continue with current practices, they can expect to continue getting the same results. Putting an increased 
emphasis on research, program evaluation, and data literacy will provide Minnesota districts with the information that 
will promote program improvement and accelerate outcomes for all students. 


ANALYSING AND REPORTING ACHIEVEMENT GAPS: GUIDANCE FOR MINNESOTA SCHOOLS 

The document entitled Analyzing and Reporting Achievement Gaps: Guidance for Minnesota Schools was recently 
released. This report emerged from a three-year collaboration between personnel at the University of Minnesota and 
members of the Minnesota Assessment Group (MAG). It provides key guidance for ongoing efforts to understand and 
reduce the achievement gap and improve educational 
outcomes. The report provide includes a concise definition 
of an achievement gap, which is an often misunderstood. It 
also provides nuanced guidance for analyses, reports, and 
communications about achievement and achievement 
gaps. That includes guidance for state and local 
educational agencies to estimate the size of an 
achievement gap in standardized units so to make the magnitude of achievement differences clear (Small > .2, Medium 
> .50, Large > .80). This illustrates a collaboration to enhance Minnesota's to use evidence and data for education. 



"Achievement gaps represent the condition 

where achievement can be predicted by 
non-academic factors, such as race, 
ethnicity, income, or neighborhood" 

(Rodriguez et al., 2016, p. 2). 


Although the report illustrates the potential of these collaborations, the process by which it was developed also 
illustrates the barriers. There was no funding or infrastructure to support the work. The lack of infrastructure and 
investment prevents many projects like this from emerging; and, if they emerge, they often do not persist. 


MIND THE GAP: MCA 2013 READING AND MATH 
STANDARDIZED EFFECT SIZE DIFFERENCES IN PROFICIENCY 
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11 http://www.act.org/newsroom/data/2015/states.html; http://www.act.org/newsroom/data/2015/pdf/profile/Minnesota.pdf; 
http://www.startribune.com/minnesota-fourth-graders-see-drop-in-math-reading-scores-on-national-exam/337784361/ 


12 MDE Report Card: : http://rc.education.state.mn.us/ 
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COMMON BARRIERS AND UNMET NEEDS 

Many people have noted that schools and districts have access to large quantities of data, but may not understand how 
to use that data to improve student outcomes; in fact, the phrase "data rich, but information poor" has been used to 
describe this situation. 13 Successful schools use data to aid their decision-making. 


DATA IMPROVES OUTCOMES 

A recent study found that five school districts that showed three years of improvement in student achievement used a 
common set of strategies, one of which was "...extensive use of data. The districts did not just talk about data; they used 
data to guide important decisions about teaching and learning." 14 

More recently, a large urban district in the upper Midwest showed a substantial positive change in high school 
graduation rates. Their success was attributed to high quality use of data and evidence-based practices. 

"Thoughtful, committed educators simply figured out what mattered and launched a groundswell of changes.... It 
began with smart research and good data. " 1S 

Slotnik & Orland (2010) stated the following: 

Improving the nation's schools requires breaking the pattern of being data-rich but information-poor. Building 
the capacity to convert data to information would give educators and policymakers the tools needed to probe for 
causes of underperformance, analyze the conditions that are contributing to varying levels of student 
achievement, and develop and implement improvement strategies based on these analyses. Teachers, 
administrators, and policymakers critically need these capacities. 

In addition, many school districts in Minnesota and around the nation are using Response-to-lntervention (Rtl) or Multi- 
Tiered Systems of Supports (MTSS) to improve outcomes for all students and close the achievement gap for at-risk 
students. MTSS/Rtl has quickly emerged as a framework to improve outcomes for all students through high-quality 
instruction tailored to student needs within a data-based decision-making model. A recent meta-analysis indicated that 
an Rtl framework, implemented with fidelity, has an effect size of 1.09. 16 This is equivalent to a learning rates 
accelerated by two-to-three years. Currently, two sources of data exist in Minnesota regarding Rtl implementation: (1) 
the 2014 Rtl Implementation Survey conducted by Wilder Research, 17 and (2) recommendations from a legislatively 
mandated Rtl task force. 

In 2014, Wilder Research, in collaboration with the Minnesota Department of Education, developed and disseminated 
the Rtl Implementation Survey to all Minnesota school districts. Over 1500 schools completed the survey, and the 


13 Slotnik, W.J., & Orland, M. (May 6, 2010). Data Rich but Information Poor. Education Week, 29 (31). 

http://www.edweek.org/ew/articles/2010/05/06/31slotnik.h29.html : Harris, S., Edmonson, S., & Combs, J. (2013). Examining what we do to 
improve our schools: 8 steps from analysis to action. New York: Routledge. 

14 Togneri, W., & Anderson, S.E. (September-October, 2003). How high poverty districts improve. Leadership, 33(1), pp. 12-16. 

15 Knowles, T. (2014, April 25). Pushing Chicago's graduation rates to new heights. Chicago Tribune. http://articles.chicagotribune.com/2014-Q4- 
25/opinion/ct-dropout-crisis-chicago-solution-perspec-0425-20140425 1 school-graduation-graduation-rate-new-heights 

16 Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning. Routledge. 

17 Gozali-Lee, E. (2014, June). Overall Results of the 2014 Minnesota Response to Intervention Implementation Survey. St. Paul, MN: Wilder 
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results found that 27% of schools reported full implementation of the framework and a substantial percentage reported 
partial implementation of the framework; however, programs that are partially or poorly implemented do not cause 
gains similar to those with full and high quality implementation. The results of this survey identified many needs in the 
area of data-based decision making (see Table 1). For example, only 12% of districts responding to the survey indicated 
that they were at full implementation with respect to using standards-based grading systems and only 19% of districts 
are evaluating the interventions they are using with subgroups of students to determine their effectiveness. Overall, the 
results of the survey point to many unmet needs in the areas of data use and decision-making. 

Table 1. Response to Intervention (Rtl) Implementation Survey Results: Data-Based Decision Making 


Survey Item 

"Fully Implementing" 
(%) 

Assessments at the lesson and unit levels are valid and reliable indicators of progress 
toward grade level benchmarks. 

30% 

Standards-based grading system that measures and monitors progress toward benchmarks 
is in place with reliable ratings between staff. 

12% 

Instructional staff understand and can communicate to parents/guardians the purposes 
and value of the assessments used, as well as their limitations. 

34% 

Data used for decision-making are accessible and timely for instructional planning. 

40% 

Multiple assessment measures are used to review the overall effectiveness of universal 
curriculum and instruction for sub-groups of students and instruction is adjusted 
accordingly. 

21% 

Supplemental interventions provided to students are evaluated for their effectiveness and 
efficiency in moving all students toward proficiency. 

29 % 

Supplemental interventions provided to students are evaluated for their effectiveness and 
efficiency in moving subgroups of students toward proficiency. 

19% 

Schools use valid and reliable tools to monitor the progress of students receiving additional 
support. 

39% 

Service providers or data teams frequently review progress monitoring data to gauge 
whether students are making adequate progress. 

34% 


In addition, a result of the 2014 Legislative Session was a legislative mandate to the Minnesota Commissioner of 
Education to develop recommendations, consistent with Minnesota Statutes, section 125A.56, for improving the 
academic performance of students through a multi-tiered system of early intervention and instructional support. As a 
result, a task force was formed to carry out the legislative mandate. Several recommendations emerged from this work. 
One, among others, was that every district should have a defined MTSS model along with technical assistance and 
support to districts . In addition, it was recommended that accountability systems within districts and schools be 
established to ensure that practices are evidence-based, culturally relevant, implemented with fidelity, and used to 
improve student outcomes and close gaps in achievement. These practices are best implemented and mostly likely to be 
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effective when they are systematically evaluated with data. Data systems and data literacy are fundamental to 
improvement. 


DATA USE AND DATA LITERACY 

Improvement occurs when valid data are collected with fidelity and used. Although a wealth of data are available and 
accountability policies have strongly encouraged the use of data to inform decisions and track progress, many studies 
have identified barriers to effective data use. 18 Professionals can learn and respond to data only if they have the 
knowledge, skills, interest, time, and access to the right types of data. Published researched indicates those 
requirements are rarely met. 19 

Barriers and unmet needs often include: 

a) Access to the right types of high quality data that are useful to guide their decisions 20 

b) Timely and convenient access to data or reports prepared by highly qualified personnel; 

c) The time to analyze and use data and evidence to inform educational decisions 21 ; and 

d) Access to experts and professional development to build the skills and knowledge use data effectively. 22 

The leadership at CAREI expected there were similar barriers and unmet needs in Minnesota. In fact, the Educator Policy 
Innovation Center (EPIC), founded by Education Minnesota, convened teachers from across Minnesota to evaluate the 
current role of testing in our state in 2015. This group reached consensus on a number of recommendations including 
supporting teachers in creating, using, interpreting, and improving a range of assessment tools. Thus, we undertook a 
statewide needs assessment designed to examine the current capacity and unmet needs of school districts and 
educational agencies in Minnesota in the areas of research, evaluation, assessment, and data use. 

PURPOSE OF THE NEEDS ASSESSMENT 

The purpose of this study was to implement a systematic process to learn from and respond to the assessment 
(collection of data) and evaluation (use of data) at all levels of Minnesota educational system. This report is the first part 

of problem solving. 

1. Problem identification and definitions: Define barriers and unmet needs in Minnesota that prevent educational 
systems and professionals from learning and responding to data; 

2. Problem analysis: Evaluate and determine what causes and maintains the problem; and 

3. Plan and propose a solution: Devise the intervention and necessary resources and propose it to those who 
allocate those resources. 

The next step is to implement and monitor the implementation of a plan/solution. 


18 Marsh, J. A. (2012). Interventions promoting educators' use of data: Research insights and gaps. Teachers College Record , 114(11), 1-48. 

19 Marsh, J. A. (2012). Interventions promoting educators' use of data: Research insights and gaps. Teachers College Record, 114(11), 1-48. 

20 Coburn, C. E., Honig, M. I., & Stein, M. K. (2009). What's the evidence on district's use of evidence? In J. Bransford, D. J. Stipek, N. J. Vye, L. 
Gomez, & D. Lam (Eds.), Educational improvement: What makes it happen and why ? (pp. 67-86). Cambridge, MA: Harvard Educational Press. 

21 Ingram, D., Louis, K.S., & Schroeder, R. G. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve 
practice. Teachers College Record, 106(6), 1258-1287. 

22 Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006). Making sense of data-driven decision making in education: Evidence from recent RAND research 
(No. OP-170-EDU). Santa Monica, CA: RAND Corporation. 
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In brief, this report summarizes the barriers and unmet needs in Minnesota along with a proposed solution to enhance 
the infrastructure and capacity to improve overcomes at reasonable cost and with substantial return on investment. 


METHODOLOGY AND DATA COLLECTION 


CAREI spent much of 2015 engaged in reviewing relevant reports, conducting interviews, and administering surveys in 
an effort to identify the unmet research, evaluation, assessment, and data needs in Minnesota's educational systems. 
We met with many key stakeholders to learn about their needs and situations. A list of individuals and professional 
organizations that participated was presented earlier in this report (Acknowledgments, p. iv). Table 2 provides a brief 
summary of the methods and data sources. 

CAREI leadership participated in meetings to interview and discuss the needs and possible solutions with the leadership 
of many state organizations. Subsequently, CAREI staff developed and administered surveys with content that reflected 
what we learned in our initial meetings. The process was iterative in that we refined our methods as we learned more 
about the needs and possible solutions that were described to us. For example, CAREI assembly districts were surveyed 
and/or leadership was interviewed early in the process. Those responses informed the development of subsequent 
surveys and interviews. In addition, various resources and reports were identified in these meetings, which we 
selectively reviewed and used to guide the study. 

Surveys were developed in collaboration with educational agencies and professional organizations (e.g., MREA, MASE, 
MASA, and DACs). Each survey was customized for the targeted group and to accommodate the preferences of the 
various groups. Although each survey was intended to yield data with similar interpretations and uses, the wording and 
composition of items varied across surveys. The results were aggregated when appropriate, and they are often 
presented together in this report to ease interpretation and use. 23 


Table 2. Data Collection Activities and Number of Participants 


Data Source 

Participants 

(N) 

Individual Interviews with Key Stakeholders and Organizations 

13 

Group Interview with Directors of Regional Centers of Excellence 

5 

Input from Member Districts of the CAREI Assembly 

16 

Discussion with Members of the Minnesota Assessment Group (MAG) 

15 

Survey of Minnesota Rural Education Association (MREA) members 

418 

Survey of Minnesota Administrators of Special Education (MASE) and Minnesota 
Association of School Administrators (MASA) 

119 

Survey of Minnesota District Assessment Coordinators (DACs) 

219 

TOTAL 

807 


23 The results from the surveys are available on the CAREI website ( http://www.cehd.umn.edu/CAREI/ ) or email (CAREI-Director@umn.edu) 
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The information obtained from the interviews and the surveys are combined and summarized according to the following 
themes of research, evaluation, assessment and data use. 


1. Educator Beliefs 

2. Unmet Needs of Educators 

3. Barriers and Capacity of Districts/Systems; and 

4. Unmet Needs of Districts/Systems. 

The results are presented in the next section. 
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A word cloud is presented above. The prominence of each word corresponds to its frequency of occurrence in 
interviews, meetings, and surveys that informed this evaluation. 

Prominently words in the cloud (above) coincide with some common themes, which emphasize the need for services 
(online tools/portals to college, manage, access, review and understand data) along with access to expertise and 
personnel (resources, staff, training, and experts) to streamline the use of data to make informed decisions. 
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RESULTS 


I. EDUCATOR BELIEFS: RESEARCH, EVALUATION, ASSESSMENT, AND DATA USE 

Survey respondents were asked to check the areas in which data reports would be helpful. In general, most respondents 
believe that quality data can improve decision-making. In our interviews and conversations, nearly everyone with whom 
we spoke was able to articulate ways in which they or their members could benefit from additional assistance. 


Educators believe that data, analysis and reports 


are useful to ... 

MONITOR THE PROGRESS OF 
INDIVIDUAL STUDENTS 

MONITOR PROGRESS IN 
MEETING DISTRICT GOALS 

TAILOR INSTRUCTION FOR 
INDIVIDUAL STUDENTS 

RECOMMEND SERVICES FOR 
STUDENTS 

EVALUATE EFFECTIVENESS OF 
SPECIFIC PROGRAMS 

IDENTIFY PROFESSIONAL 
DEVELOPMENT NEEDS 

MEET ACCOUNTABILITY 
REQUIREMENTS 


ASSIGN STUDENTS TO CLASSES 


INFORM SCHOOL OR DISTRICT 
POLICIES 

DATA REPORTS WOULD NOT 
HELP 



1. Data to monitor and improve outcomes: Over 70% of 
the respondents indicated that data reports would be 
helpful to monitor individual student progress (77%), track 
system improvement (75%), tailor instruction for 
differentiation (73%), develop instructional 
recommendations (71%), evaluate school and district 
programs (70%), and identify professional development 
needs (70%). 



"We need more professional development 
around using data from tests and having 
that impact instruction and interventions." 


2. Data are useful: Almost all respondents (98%) 
disagreed with the statement "data reports would not be 
helpful." Leaders and educators believe in the use of data. 

3. Limited awareness for data use: Only about half of 
respondents indicated data reports are useful to assign 
students to classes or groups (52%) or inform school or 
district policies (47%). There is a need to increase 
awareness for the use of data and reports. 



"We have lots of data, but we struggle with 
helping teachers use the data to drive 
instruction effectively." 


Figure 3. District and School Leaders: Usefulness of Data Reports 


Expert Insights 

In the interviews, several stakeholders suggested that CAREI work with the SLEDS (the MN Statewide Longitudinal 
Education Data System) database to provide information about student pathways from kindergarten, through 
enrollment in and completion of post-secondary education, to employment. 
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II. UNMET NEEDS OF EDUCATORS: RESEARCH, EVALUATION, ASSESSMENT, AND DATA USE 


Survey results identified critical barriers and unmet needs. Educators need more timely access to data, professional 
development, time to use data, and access to prepared reports to use data. They also require expert assistance for 
training, preparation of reports, and support to use the reports to guide important educational decisions. 


Educators report they need... 


DATA MADE AVAILABLE IN A 
TIMELY MANNER TO HELP 
INFORM A DECISION 

SUFFICIENT TIME TO USE DATA 
EFFECTIVELY 

PROFESSIONAL DEVELOPMENT 
OPPORTUNITIES TO LEARN 
MORE ABOUT HOW TO USE... 

SOMEONE WHO CAN PROVIDE 
CUSTOMIZED DATA REPORTS 
FOR THE SPECIFIC NEEDS OF... 

SOMEONE STAFF CAN GO TO 
WHO CAN ANSWER 
QUESTIONS ABOUT USING- 

HELP WITH EVALUATING 
EXISTING ASSESSMENTS* 


STANDARD PROTOCOLS FOR 
EXAMINING DATA* 


HELP WITH CREATING STUDENT 
ASSESSMENTS* 

COMPUTER SYSTEMS THAT 
PROVIDE ACCESS TO LOTS OF 
DATA 


% in Need 


72% 


71% 


69% 


61% 


59% 


50% 


49% 


43% 


32% 


*Not included on the MN Rural Education Association Survey. 


4. Timely access to data: 72% of educators need more timely 
and immediate access to data; 71% need more time to use 
data. 


"We 
to e 


"We are lacking time built into our schedule 
to examine and discuss data." 


5 Professional development and support: 69% of educators 
need additional training and support to use data; 61% would 
like someone to help with the analysis of data and the 
creation of useful reports; 59% need access to more 
knowledgeable staff or experts; 49% indicate a need for 
standards and processes for interpretation and use of data. 


"We need more professional development around 
using data from tests and having that impact 
instruction and interventions." 

"[We have] a lot of information but have difficulty 
organizing that information." 


6. Assessment and evaluation: 50% of educators need 
assistance to understand and evaluate the assessments they 
use; 43% need assistance to develop new assessments. 


Figure 4. District and School Leaders: Unmet Needs 


Expert Insights 

more support must be given to teachers to maintain and develop their ability to create, use, interpret, and improve 
the full range of assessment tools." (EPIC, page 20). 
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III. BARRIERS AND CAPACITY OF DISTRICTS/SYSTEMS: RESEARCH, EVALUATION, ASSESSMENT, AND 
DATA USE 

Members of MASE, MASA, and the district assessment coordinators (DACs) were asked to rate their district's capacity 
(Very Good, Good, Fair, Poor) for research, evaluation, assessment, and data use. The largest percentage of respondents 
indicated their capacity was fair or poor across all areas. 


7. Professional development and support: The majority 
of educational leaders believe their capacity for training 
and support for high quality interpretation (70%) and 
use of data (66%) was either fair or poor. 


8. District assessment programs: District and school 
administrators need more capacity to develop surveys 
for staff and students (66% Fair/Poor), monitor student 
progress (57%), select diagnostic measures (57%), and 
select screening measures (48%). 

9. Data use: 54% of districts administrators lack capacity 
to analyze their data, and 51% lack capacity to report 
their screening and other types of data. 

10. Personnel and costs (not depicted in the figure): 
Only 33% of administrators have staff with advanced 
training (MA, PhD) in research, evaluation, assessment 
or data use because qualified personnel are not 
available, (b) they are too costly, or (c) there is 
insufficient funding. 


11. Program evaluation: 51% of administrators indicated 
their capacity to evaluate policies and programs was 
poor. High quality program evaluation is infrequent 
because of "lack of time" (78%), inadequate 
staffing/expertise (63%), or cost (53%). 

Figure 5. Capacity for Research, Evaluation, Assessment, and Data Use 


Districts and schools need access to experts 
"for research, evaluation, assessment, and 
data-based decision making." 


"CAREI could provide— help with questions, 
help with the model, help with technology." 


District and school capacity was rated by administrators as 
insufficient (Fair/Poor) for key functions to use data. 


PROVIDE TRAINING ON HOW TO 
INTERPRET DATA 

PROVIDE TRAINING ON HOW TO 
USE DATA TO MAKE INFORMED 
DECISIONS 

DEVELOP QUESTIONNAIRES AND 
SURVEYS FOR STUDENTS, 
PARENTS, TEACHERS, AND/OR.. 

USE DATA TO MAKE RESOURCE 
ALLOCATION DECISIONS (E.G., 
STAFFING, TRAINING,... 

PROVIDE INFRASTRUCTURE IN 
THE AREA OF STUDENT 
PROGRESS MONITORING 

SELECT DIAGNOSTIC 
ASSESSMENTS 

ANALYZE DATA FROM 
SCREENING/BENCHMARK 
ASSESSMENTS 

EVALUATE DISTRICT OR 
BUILDING PROGRAMS AND 


ANALYZE DATA AND PREPARE 
SUMMARY REPORTS FOR 
TEACHERS, TEAMS, OR... 

SELECT SCREENING/BENCHMARK 
ASSESSMENTS 

FULFILL REQUESTS FOR DATA 
FROM TEACHERS OR 
ADMINISTRATORS 



100 % 
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12. Leaders and policy makers routinely adopt and implement programs. These programs are rarely evaluated because 
78% of educational leaders lack the time, 63% lack staff, and 53% lack resources (Figure 6). 



LACK OF TIME INADEQUATE STAFFING LACK OF FINANCIAL NO ACCESS TO EXTERNAL 

RESOURCES EVALUATORS 


Figure 6. Survey Results: Barriers to Program Evaluation 


Expert Insights: District Capacity 

it Research, Evaluation and Assessment Departments: "they're few and far between ... All the big districts 
have that. Medium and small districts may have an individual or department, or are working to add it." 

(Scott Fitzsimonds, Data Specialist, Regional Centers of Excellence) 

■fr "Smaller districts don't have the staff." 

(Directors, Regional Centers of Excellence) 

it The six regional centers share one staff position to support data analysis and use. 

(Directors, Regional Centers of Excellence) 

it Districts need people with "...the technical expertise to do research and evaluation. That varies a lot from 
district to district. Districts that have that staff do more of it obviously. Districts that don't have it probably 
don't do any but they certainly need it." 

(Jim Angermeyr, retired REA Director, Bloomington) 

it "Everybody seems to have an assessment person or persons who administer the tests, but it seems to be 
less common that you have a person that has the skills and background to do the evaluations, to even make 
presentations to the school board that are beyond the very descriptive kind of analyses. So I think there's a 
great need..." 

(Dave Heistad, REA Director, Bloomington, and former Director, Minneapolis Public Schools) 

it "I really started thinking about a state-wide vision when I worked with the Bush Foundation ... These small 
districts that came in and received value-added reports had a chance to see the strengths and weaknesses 
of their schools. They really just ate it up. I think you can even go from ... more state accountability 
assessments, down to response to intervention programs to survey research to providing aggregate reports 
like school information reports. They just don't have the infrastructure to do any of that." 

(Dave Heistad, REA Director, Bloomington, and former Director, Minneapolis Public Schools) 
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IV. NEEDS OF DISTRICTS/SYSTEMS: RESEARCH, EVALUATION, ASSESSMENT, AND DATA USE 


Survey respondents (MASE/MASA and DACs) and interviewees provided information on the unmet needs that schools 
and districts have in the areas of research, evaluation, assessment, and data use. Results are summarized below. 

TIME. As mentioned above, one of the most prevalent needs that people listed was the need for more time to work 
with data. Examples of comments from survey respondents include: 

Time for utilization of data. 

■S Restructuring time for staff to complete mindful research , evaluation and assessments in order to develop data 
based decisions. 

V Our district data team needs to create a time to meet consistently. 

■S We lack time that is built into to our schedule to examine and discuss data. 

TRAINING. Survey respondents and interviewees indicated a need for more training on how to analyze, interpret, use, 
and display data. This could include the use of common templates or protocols for examining data. Examples of 
comments from survey respondents include: 

V We need more professional development around using data from tests and having that impact instruction and 

interventions. 

■S Help in using the data provided from testing. 

V We need our teachers to receive professional development on data interpretation and instructional match. 

V We have lots of data, but we struggle with helping teachers use the data to drive instruction effectively. 

V a simple protocol to collect, manage, and relay meaning of data would be a good place to start 

STAFF EXPERTISE. Survey respondents and 
interviewees acknowledged that districts have varying 
levels of staff expertise to conduct research, 
evaluation, and assessment tasks. Survey respondents 
felt that an unmet need of theirs was staff expertise 
and resources for staffing in this area: 

V Respondents indicated a need "for research, 
evaluation, assessment, and data-based 
decision making. " 

■S Another indicated that they need "Resources to 
hire staff to lead in research, evaluation and 

assessment." 

V A District Assessment Coordinator said they 
could use the support of "a data expert who 
could attend regular data meetings with staff. " 
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Expert Insights: 

"...districts have people that manage the testing 
program, but not very many people trained as 
researchers or with assessment backgrounds. 

We have a lot of programs that train principals and 
even programs that train superintendents, but I think 
the assessment role is pretty much catch as catch 

can . Which is another thing that CAREI could do with 
its connection to the university." 

(Jim Angermeyr, President, Educational Assessment 
and Evaluation Services, LLC) 
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DATA ORGANIZATION. Survey respondents indicated that they could benefit from data organized in a useful manner. 
Having a uniform system across school districts was also mentioned as something that would enhance the use of data in 
school districts. Examples of the responses from MASE/MASA and DAC survey respondents include: 

•S Simple access to the data in a manner that can be used to make decisions. 

■f Ability for staff to easily pull ... data. 

■S Data bank that is easy to display and not something that takes so much to figure out. 

■S Streamlined systems that are not cumbersome to access data. 

■f Online data portals where all student data is kept, and is transferable from school to school. 

•S No uniform system for student management statewide! This would allow import and export of data between 
school districts and for comparison between districts... I believe there would be a huge savings in time and 
resources to get us all moving in the same direction on student information and data. 

ASSESSMENT DEVELOPMENT. In the interviews, we heard people talking about how districts need help to use 
formative and benchmark assessments. We also heard that districts struggle with selecting assessments for their 
district assessment program. "There are standards for assessments, and we could help districts if they're in the 
process of selection to look at reliability and validity standards, and content matched with standards" (Dave Heistad, 
Executive Director of Research, Evaluation & Assessment, Bloomington Public Schools). 

Survey respondents also discussed the need for continued help in developing and selecting assessments. Examples 
of comments from survey respondents include: 

S Continued support on creating assessments aligned to standards. 

S We need help guiding teachers to create reliable formative assessments. 


CARETS CAPACITY TO RESPOND TO UNMET NEEDS 


Our findings suggest there are many needs across the state that CAREI could meet. It already has the infrastructure, 
networks, expertise and experience to coordinate services, especially for research, evaluation, assessment, and data 
use. 

CAREI seeks to provide statewide support to policy makers, state educational agencies, and local educational agencies. 
CAREI can provide research, evaluation, assessment, and data services for state agencies and the more than 2,000 public 
schools in the state of Minnesota, which serve over 800,000 students. Our work is just beginning and nearly half (48%) 
of the survey respondents indicated they would likely access services from CAREI. That number will increase 
substantially as the work and services of the New CAREI become known across the state. 


PROPOSED SOLUTION 


As a college-wide center at the University of Minnesota (University), The Center for Applied Research and 
Educational Improvement (CAREI) is well-positioned to coordinate efforts and deliver services that meet the 
needs of the more than 2,000 public schools in the state of Minnesota, which serve over 800,000 students. Many 
of the school districts do not employ staff with the necessary expertise in research, evaluation, assessment, and 
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data use. As a result, key methods and processes of data use do not occur, which substantially limits Minnesota's 
capacity for systematic improvement and learning from data. CAREI can be the entry point and facilitator for policy 
makers and educational leaders to access faculty, staff, resources, and efficiencies at the University to bring 
systematic processes and data to life. Learning to improve with data requires collaborative networks among policy 
makers, practitioners, researchers, evaluators, and psychometricians. 

CAREI will coordinate and maintain Networked Improvement Communities (NICs) 24 ' 25 to facilitate and maintain 
high quality collaboration and communications among policy makers, educators, researchers, evaluators, and 
psychometricians. As part of the NIC process, CAREI will serve as a "network hub" to provide training, support, and 
data analytic services as our school networks engage in trial-and-error solution generation and implementation to 
move targets to meet their goals. We hope that school districts working together in a networked fashion will allow 
them to quickly learn about what works, modify implementation as needed, and learn to improve through shared 
results and information. CAREI will support networks by helping them develop their values, goals, and priorities 
and then developing tools to measure progress. CAREI will assist with disseminating information about what does 
and does not work to support improved practice. CAREI will coordinate systematic and focused processes 
targeted at key challenges at the state and local level. CAREI will assist to develop, compile, disseminate, and 
automate materials and services to build efficiencies among collaborators and agencies (e.g., automated analysis 
and reporting of data, surveys, formative assessments, professional development, literature reviews). CAREI will 
provide these services to policy makers, the Minnesota Department of Education, districts, schools, and 
educational professionals; the services will be available at a reasonable cost to leverage a high return on 
investment. It is much more cost effective and efficient to centralize these services where the expertise already 
exists within the University. Collaboration and services to educational agencies will focus, and make more 
effective, state and local investments to improve educational outcomes. In addition, CAREI will continue to provide 
more intensive and specific services to agencies, which include school-based evaluations that require onsite data 
collection and collaborations with agencies to compete for federal funding of educational programs. 

As a low cost nonprofit with an established infrastructure (e.g., University salaries, offices, telecommunication), 
the value proposition and return on investment for state and local agencies is substantial relative to that of the for- 
profit agencies and Washington beltway firms who currently provide many of these services. 


SUMMARY 


Although the state of Minnesota currently has the highest ACT scores in the country, it also has the largest achievement 
gap between students of color and White students. 26 In addition, statewide accountability tests indicate that 
approximately 40% of students have not met grade level standards in reading and math. The results of this needs 


24 Bryk, A.S., Gomez, L.M., Grunow, A., & Lemahieu, P.G. (2015). Learning to Improve: How America's Schools Can Get Better at Getting Better. 
Harvard Education Press: Massachusetts. 

25 Bryk, A.S., Gomez, L.M., Grunow, A. (2010). Stanford, CA: Getting Ideas into Action: Building Networked Improvement Communities in 
education. Carnegie Foundation for the Advancement of Teaching, http://www.carnegiefoundation.org/resources/publications/getting-ideas- 
action-building-networked-improvement-communities-education/ 

26 http://www.startribune.com/minnesota-s-achievement-gap-debated-at-nabi-conference/321150101/; 

http://www.education.state.mn.us/MDE/Welcome/QfficeCom/CloseAchievGap/056710 ; http://minnesota.cbslocal.com/2013/Q8/21/good- 
question-why-such-a-large-achievement-gap-in-mn-classrooms/ 
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assessment confirm that school districts are in need of professional development and technical assistance in the areas of 
research, evaluation, assessment and data use. This will assist them in ensuring adequate progress for all students. A 
previous needs assessment 27 found similar results, and a legislatively mandated task force on MTSS (multi-tiered 
systems of support) had similar recommendations as to those suggested here. 28 

Overall, the survey and interview data collected for this CAREI needs assessment suggest that although many districts 
are doing excellent work in the areas of research, evaluation, and assessment, there are also still many unmet needs as 
documented in our findings. The results of this needs assessment suggest four conclusions: 

1. Respondents to our surveys and interviews believe that high-quality data can improve their decision- 
making. 

2. There is a large need for increased "data literacy" at all levels including teachers and administrators. 
Schools are data rich but information poor. 29 An overarching theme was that our school staff and 
leaders need to understand how to use data for various types of decisions related to student outcomes. 
They are not always sure what data to collect or what pieces of information are worthy of their 
attention. 

3. Districts currently lack capacity in the area of data-based decision-making for a wide variety of reasons, 
mostly centered on training and a lack of staff with expertise in research, evaluation, and assessment. 
Districts do not have the resources to meet all the research, evaluation, and assessment needs that they 
have due to lack of time, lack of professional development in these areas, lack of personnel trained in 
these areas, and lack of resources. 

4. A need exists to expand services in the area of research, evaluation, and assessment so that all school 
districts in Minnesota have the technical assistance they need to improve educational outcomes for all 
students. 

CAREI is well-positioned to address the statewide gaps in the areas listed above. CAREI will be able to provide the 
infrastructure needed in our state in the areas of research, evaluation, and assessment. We at CAREI intend to leverage 
quality services to school districts through collaborative efforts with key stakeholders in our state. We all agree on a 
common mission and that is to improve outcomes for all of Minnesota's students. 


"We must demand better. We must demand a system that expects teachers to regularly assess and provide 
high-quality feedback to their students, and one that supports teachers in developing and maintaining the 
many skills needed to do so. We must demand a system that values quality of state testing over quantity, 
while addressing the very real crisis of achievement gaps. We must demand a system that empowers schools 
and their surrounding communities to work together to define goals that include, but also transcend those 
established by the state, and to work together in pursuit of those goals." (EPIC, 2015. Testing Better: How to 
Improve Minnesota's Use of Assessments in Education. P.36) 


27 Gozali-Lee, E. (2014, June). Overall Results of the 2014 Minnesota Response to Intervention Implementation Survey. St. Paul, MN: Wilder 
Research. 

28 Wilder Research. (2014, November). 2014 Implementation Portrait of Multi-tiered System of Supports and English Language Arts Standards. 
Presentation given to MTSS Task-Force, MN. 

29 Slotnik, W.J., & Orland, M. (May 6, 2010). Data Rich but Information Poor. Education Week, 29 (31). Retrieved from: 
http://www.edweek.org/ew/articles/2010/05/06/31slotnik.h29.html. 
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